Automated operation of railroad trains

ABSTRACT

A new over lay technology for the Positive Train Control and Energy Management systems used by the railroad industry today which allows for automated train handling responses to potential on-track hazards. Various sensors, including image-capture devices, radar, and drones, are placed on or proximate to a train. These sensors are used to interface with or override the Positive Train Control and Energy Management systems where those systems activate specific actions such as slowing or stopping a train but hazardous conditions on the track may dictate an alternative response.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional patent applicationNo. 63/330,110, filed Apr. 12, 2022. The entirety of that provisionalapplication is incorporated in its entirety herein by reference.

FIELD OF THE DISCLOSURE

The present disclosure relates to the field of automated operation ofrailroad trains.

BACKGROUND OF THE DISCLOSURE

The Rail Safety Improvement Act of 2008 mandated the implementation of afederal railway safety system called Positive Train Control (PTC).According to the U.S. Department of Transportation, PTC systems “aredesigned to prevent train-to-train collisions, over-speed derailments,incursions into established work zones, and movement of trains throughswitches left in the wrong position.” Positive Train Control (PTC), U.S.DEPT. OF TRANSP., FED. R.R. ADMIN.,https://railroads.dot.gov/train-controUptc/positive-train-control-ptc(last accessed Mar. 11, 2022).

PTC complies with the 2008 Rail Safety Act by monitoring the current andpredicted operation of a train and providing warnings to the locomotiveengineer that action is required to maintain safe operation. If thelocomotive engineer fails to adequately respond to PTC warnings, PTCwill stop the train with a penalty brake application.

A typical PTC system includes a monitor for the locomotive engineerwhich displays distances to next speed and stop targets with which thelocomotive engineer must comply. These targets are established as PTCnavigates against a detailed track database using GPS and dead reckoningtechniques. The detailed track database includes precision locations ofwayside signal assets, precision locations of critical track featuresand all civil speed restrictions. Additionally, wirelessly communicatedwayside signal indications and dispatch office mandatory directives areused by PTC to create speed and stop targets.

While the PTC system was being implemented, the railroad industrycontinued the development and implementation of Energy Management (EM)systems as well. EM systems can vary by individual rail company, but EMsystems preserve train momentum to reduce fuel consumption whilemaintaining on time train performance. Other benefits include reduced intrain forces and improved train handling. These automated systems caninclude throttle control and, application of pneumatic and regenerativebrake systems based on track topology, route data, operatingconstraints, and consist information for individual trains.

One of the ways that PTC affects the speed of trains is by creatingRestricted Speed targets by rule or in response to wayside signalindications, track data or mandatory directives. When operating within aRestricted Speed target, a human engineer is required to maintain aspeed such that the engineer can stop the train within one-half therange of the engineer's vision. PTC cannot enforce this one-half rangeof vision requirement. This leads to inefficient and slow railoperations because engineers naturally are conservative with theirestimates rather than risk a collision. Yet Restricted Speed collisionsdo occur and although at slower speeds, they may still have catastrophicconsequences.

Currently PTC and EM rely on static track databases which contain grade,curvature, civil speed and critical feature location (i.e. switch andsignal) information. PTC and EM are not equipped to handle many of thedynamic conditions that may occur on the railroad right of way. Whilevigilant train engineers can serve to lessen this gap, detectioncapabilities of advanced sensor packages can reduce and/or mitigatehuman error, particularly where operator fatigue or poor visibility arefactors.

Accordingly, what is needed is a step beyond PTC to allow trains toautonomously respond to dynamic and/or changing conditions on the track.

SUMMARY

This disclosure is intended to advance the automation of trains. Thepurpose of devices and methods disclosed herein is to makebest-practice, consistent responses to objects on or approaching thetracks which surpass human responses in both consistency and vigilanceto achieve safer outcomes. The systems and methods disclosed herein maybe used together with one or more human operators onboard the train oron a train with no human operator.

The present disclosure, directed toward a new train monitoring systemutilizing computer vision, provides automated response to line-of-sighthazards on the right of way and provides an evolutionary path to betterthan line-of-sight detection. The present system improves computerfunctionality and computer vision for purposes of rail autonomousoperations with or without a human operator, the underpinnings of whichare center in the creation of Baseline Navigational Data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view illustrating the concept of operationsaccording to an embodiment.

FIG. 2 is a block diagram illustrating a system architecture of anembodiment.

FIG. 3 is a schematic diagram representing an exemplary Impact Zoneaccording to an embodiment.

FIG. 4 is a schematic diagram illustrating an exemplary UnobstructedDistance in an embodiment.

FIG. 5 is an image illustrating a car in an Impact Zone in anembodiment.

FIG. 6 is an image illustrating out of tolerance pixels corresponding tothe car in the Impact Zone.

FIGS. 7A-7B are images of the car of FIG. 5 at different locationsrelative to the tracks.

FIG. 8 is a schematic diagram illustrating a Last Trapezoid Definitionaccording to an embodiment.

FIG. 9 is a schematic diagram illustrating a Penultimate TrapezoidDefinition according to an embodiment.

FIG. 10 is a schematic diagram illustrating Trapezoidal Data Creationaccording to an embodiment.

FIG. 11 is a schematic diagram illustrating a Weighted Pixel Grid Boxaccording to an embodiment.

FIG. 12 is a schematic diagram illustrating trapezoid regions along acurved track according to an embodiment.

FIG. 13 is a flow chart illustrating a Train Handling and HazardResponse process according to an embodiment.

FIG. 14 is an image used for Baseline Navigational Data creation in anembodiment.

FIG. 15 is an image of an Active View corresponding to BaselineNavigational Data in FIG. 14 .

FIG. 16 is a justified active view image corresponding to the image ofFIG. 15 .

FIGS. 17A-17E are schematic diagram illustrating successive trapezoidsprocessed to create baseline navigation data in an embodiment.

FIG. 18 is a schematic diagram illustrating a view of an adjacent trackwith a train traveling in an opposite direction.

FIG. 19 is an exemplary trapezoid with superimposed grid boxes whichprovide a framework for pixel analysis across consecutive images.

DETAILED DESCRIPTION

Detailed descriptions of examples and methods of the present disclosureare provided below. The description of preferred and alternativeexamples, while thorough, is meant to be exemplary only, and variations,modifications, and alterations may be apparent to one of ordinary skillin the art. These examples do not limit the breadth of the disclosure.

Definitions

Active Navigation: disclosed system's processes for synchronizingBaseline Navigational Data to the active view and for declaring ahazardous condition.

Active View: the current view from the locomotive image capture deviceand other sensors while the disclosed system is operating.

Baseline Navigation Data: disclosed system's developed data used byActive Navigation to orient the system to areas of interest in andaround the track and provide data for comparison with the active viewfor hazardous condition detection.

Consist: the number of cars, length and weight of a train.

Energy Management (EM): An existing fuel efficiency system installed onmost line of road locomotives which adjusts throttle settings andprovides braking instructions to preserve momentum to conserve fuel.

Emergency Brake Application: applies maximum braking force (20% morethan full service) as quickly as possible by opening brake pipe pressureto atmosphere, once applied it is not recoverable until the train comesto a stop.

Fail-Safe: a design principle that in the event of a failure, the systemtakes the safest course of action, even if it is more restrictive thannecessary.

Hazardous condition: a condition in which any object or obstruction inthe impact zone endangers the train's safe passage and/or people and/oranimals are in danger of being struck by the train.

Impact Zone: area in and around the track where a train would impact anobject, obstruction, individual or animal.

Image Subtraction—computer operation whereby pixels from one image aresubtracted from another to detect changes between the images

Justification—a process for adjusting pixel values in the active viewbased on differences between sampled pixels represented in BaselineNavigational Data and those same pixels in the active view.

Neural Network: a subset of machine learning which must be trained toclassify objects by analyzing input data using filtered layers whichidentify object characteristics.

Object Tracking: a computer vision activity where object movement isassessed between captured images.

Penalty Brake: a full-service or emergency brake application applied byPTC which cannot be recovered until after the train has stopped.

Performance Data and System Analytics: disclosed system's processes andmethodologies to provide automated review and system learning fromoperational data.

Pixel Calculus: weighted and justified pixel summations for thedisclosed system's prescribed regions around the track for impact zoneand warning zones which in preferred embodiments are generallytrapezoidal in shape; in some embodiments, some or all of the regionsare further divided into analytical units referred to herein as gridboxes.

Pixel Signature: a collection of pixels identified as a hazardouscondition, from a pixel calculus variance representing a differencebetween the Baseline Navigation Data and Active Navigation which may beused by the disclosed rail computer vision system to monitor and updatethe status of a hazardous condition.

Positive Train Control (PTC): a Federally, mandated train safety systemwhich uses full-service and emergency brake applications to stop a trainthat is not in compliance with different operating parameters.

PTL: Positive Train Location, is a rail industry initiative to enhancePTC navigational location accuracy ensuring PTC is accurately localizedfor initialization and operation in multi-track areas by utilizing DGPS(Differential Global Positioning System), locomotive tachometer and anIMU (Inertial Measurement Unit).

Recoverable brake—train brake which does not require a stop; train brakeother than penalty or emergency brake

Restricted Speed: railroad operating rule that requires stopping thetrain within one-half the range of vision, also imposes a maximumoperating speed typically 20 mph or less.

Severity index: value assigned by the disclosed system to differentiateuse case scenarios and facilitate appropriate train handling responsesto hazardous conditions.

Subdivision: sometimes referred to as a district, a railroadorganizational designation for a line segment which typically definestracks and operations between two or more terminals or junctions.

Train Handling and Hazard Response: the disclosed system's processes forcontextualizing rail computer vision and initiating speed reduction andbraking events.

Trapezoidal Ranging: inherent byproduct of disclosed system's navigationthat yields persistent distance to hazardous conditions values.

Unobstructed distance: measured value as determined by reflectivetechnology which captures the distance to a potential obstruction.

Warning zone: area outside the impact zone where the present system willassess the need to warn people and/or animals of an approaching train.

Wayside Equipment Detector: devices used to assess the mechanical healthof a train as rolling stock pass by the device.

Exemplary Concept of Operations

FIG. 1 is a high-level exemplary diagram of the Concept of Operations(CONOPS) for the disclosed system. In an abstract example, train 101 maybe approaching hazardous condition 103. Hazardous condition 103 may beanything affecting the operational safety of train 101 which potentiallycould cause harm to the public, rail personnel and\or disrupt railoperations. By way of non-limiting examples, a hazardous condition mayinclude the following when located within the impact zone: humans(including adults and children), animals (small and large, domestic orwild), vehicles, fallen trees or telephone poles, trash or garbage, farmequipment, earth construction equipment (e.g., backhoes and earthmovers), track faults, rockslides, buckled track, washed out track,grade crossing warning device failure, misaligned switch/switch targets,improperly positioned derail devices, downed power lines, high water,on-track equipment including rolling stock and locomotives. These sameobjects located in the warning zone are not considered hazardousconditions and only elicit warning responses from the present system.

As train 101 approaches hazardous condition 103, two distances becomeimportant. The first is braking distance 102. This is the distancerequired for the train to come to a complete stop when a penalty brakeapplication is applied. PTC already makes these calculationscontinuously based on current speed, train consist, grade, trackcurvature and possibly other factors.

The second distance is response distance 104. This represents thedistance by which train 101 must complete the execution of its responseto the hazardous condition 103. For example, if hazardous condition 103is a downed tree across the track, then response distance 104 representsthe distance the system has to evaluate the hazardous conditions andbring train 101 to a stop.

If the response distance is greater than the braking distance, thereexists an evaluation distance 105, a time interval within which thedisclosed system is afforded the opportunity to further evaluate thehazardous condition. Under such circumstances, the response distanceequals the braking distance plus the evaluation distance.

The disclosed system's dynamic detection of, and responses to, theexamples of hazardous conditions discussed above, which are unaddressedby PTC or EM operational scenarios, is presented herein. Set forthherein are autonomous operation processes and methodologies whichaddress the unique requirements of rail. Introduced is highly developednavigational data which leverages the unique characteristics of railoperations and provides increased detection and response capabilities ascompared to a locomotive engineer. In some embodiments, the disclosedsystem will interface with PTC, EM and PTL through plannedIndustry-defined protocols. Table 1 below represents an evolutionarypath by which this system may be implemented and incrementally improvedwith technology.

TABLE 1 Functionality Operation Hazardous condition detection Manned bylocomotive engineer able to and classification equivalent intervene tolocomotive engineer Hazardous condition detection Unmanned locomotive -single and classification exceeding monitoring session with onboardlocomotive engineer capabilities sensors, optional operator capable ofintervention Hazardous condition detection Unmanned locomotive - dualand classification exceeding monitoring sessions for onboard andlocomotive engineer capabilities/ extended vision drone sensors,optional Strategic drone support operator capable of interventionhazardous condition detection Unmanned locomotive - multiple withclassification exceeding monitoring sessions for onboard, locomotiveengineer capabilities/ extended vision drone and inspection Continuousdrone support drone sensors, optional operator capable of intervention

The present disclosure makes no claim related to image capture devicetechnology. A combination of different image capture device technologiesmay well provide the best solution. Advances in technology will onlyserve to enhance the disclosed system's use of Baseline NavigationalData and its novel approach to a rail computer vision system whichseparates object detection and object classification so that objectdetection performed against baseline data represents a “no hazardouscondition present” truth which authorizes train operations to continueuninterrupted.

FIG. 2 is a block diagram representing an embodiment of the invention.The functions inside the dashed line in FIG. 2 may be performed on acomputer systems in a distributed manner among a plurality of computersystems. In an embodiment, a computer system includes one or moreprocessors, accompanying memory for program and data storage, andcommunications interfaces for communicating with, e.g., devices 209,210, 211, 212, data storage device(s) 201, EM and PTC systems 207, 208,and performance data and system analytics 206. Referring to theexemplary embodiment of FIG. 2 for a high-level system overview,Baseline Navigational Data is inputted and synchronized with ActiveNavigation. Baseline Navigational Data effectively provides a roadmapwhich “tells” Active Navigation where to “look” and what it should “see”if no hazardous conditions are present.

Active Navigation may use multiple locomotive capture devices andsensors to support the disclosed system's unique approach to reliablehazardous condition detection which is accomplished exclusively againstBaseline Navigational Data. The disclosed system is only required todetect an out of tolerance condition with Baseline Navigational Databefore declaring a hazardous condition, without incurring the processingrequirements imposed by a Neural Network or image subtraction. Oncedeclared, the hazardous condition will be treated with the highestseverity unless released by the disclosed system's specificallyrail-trained Neural Network. This release would typically need to occurwithin the recognition and reaction time of the locomotive engineerunless current braking distance affords the system additional evaluationtime. The efficiencies of declaring a hazardous condition in thismanner, against Baseline Navigational Data without a need for objectclassification establishes a standard of defaulting to the safest courseof action which befits the interests of public safety as it relates torail operations.

Baseline Navigational Data is organized by precise location and containspixel calculus and unobstructed distance values consistent withoperating conditions when no hazardous conditions are present. Thesevalues contained in Baseline Navigational Data can be used forcomparison with the disclosed system's active view calculated pixelcalculus values and unobstructed distance values measured during ActiveNavigation to determine if it's safe for a train to continue unabated.Baseline Navigational Data provides the necessary information for ActiveNavigation to make these determinations and clear a train for continuedoperations on a section of track.

The components of Baseline Navigation Data which focuses the active viewonly on the areas for which the disclosed system provides protection arethe impact zone and the warning zone. FIG. 3 is a depiction of exemplaryimpact zones as defined by (x,y) coordinates contained in BaselineNavigational Data which form trapezoids extending in front of the trainalong its path when applied to the active view. These trapezoids may befurther divided by the present system by superimposing grids on eachtrapezoid. Baseline Navigational Data contains values calculated overthe pixels in each of these grids/trapezoidal divisions when nohazardous condition is present, effectively telling Active Navigationwhat it should “see” with no hazardous conditions present. ActiveNavigation can now compare its own active view pixel calculuscalculations with those stored in Baseline Navigational Data todetermine whether a hazardous condition should be declared. In someembodiments, this comparison constitutes determining the differencebetween the pixel calculus values for a region stored in the BaselineNavigation Data and determined by the active view (the active view pixelcalculus values are preferably justified as explained further below). Adisclosed system's proof of concepts (POC) was conducted, the results ofwhich are discussed later in this text. The aim of the POC was todemonstrate that the baseline of a railway roadbed provides anextraordinarily consistent background for hazardous condition detectionwhen leveraged by the disclosed system's pixel calculus processes.Comparatively, conventional computer vision without the BaselineNavigational Data roadmap would need to use resources to process muchmore of the entire image presented to its computer vision system.Conventional computer vision systems without the aid of BaselineNavigational Data would presumably need to reference stored images oridentify image characteristics by passing filters over the entire imageto first discover the rail and establish an area of interest around therail, and then continue processing the image to identify any potentialhazards.

The second component of Baseline Navigational Data used to authorizecontinued operation into a section of track is unobstructed distance,which represents the minimum distances the track must be clear of anyobstruction. At prescribed locations indicated in the BaselineNavigation Data, Active Navigation performs these measurements forcomparison with Baseline Navigational Data values. Reflective technology(e.g., radar, laser) can provide reflected distances which ActiveNavigation can interpret as inside or outside the confines of the impactor warning zones by comparing the current reflected measurements withthose in Baseline Navigational Data. FIG. 4 depicts the application of areflective technology device which produces reflective distancemeasurements which may be further enhanced by multiple antennae capableof different beam widths and multiple receivers. In this depiction, thereflective technology device is focused on the outer limits of sightdistance. In some embodiments, the width of the beam 402 generated bythe reflective technology is approximately one degree. Wider or narrowerbeam widths are used in alternative embodiments. Active Navigationcalculates and compares reflective technology values for unobstructeddistance with those captured in Baseline Navigational Data, again todetermine if the locomotive can continue unabated. Unobstructed distancemay also leverage the present system's trapezoidal framework tointerpret reflected results, this may be particularly useful if thechosen technology is imaging radar.

The above-described leveraging of Baseline Navigational Data by ActiveNavigation successfully shifts a significant portion of computer visionresource requirements from the active system to an offline process inwhich Baseline Navigational Data is created. Offline computational datafrom the lab embedded in the Baseline Navigational Data file facilitateshazardous condition detection by Active Navigation.

Continuing with FIG. 2 , the impact of Baseline Navigational Datacontinues after a hazardous condition is identified by ActiveNavigation. The Neural Network classifies the object, for example, anobject may be classified as a person, an animal or a vehicle. However,by using Baseline Navigational Data as a screening process for theNeural Network, a >95% reduction in pixel count for image classificationis achievable since only the out of tolerance pixels (those pixelscorresponding to a grid in an impact zone or warning zone for which thedifference of the Baseline Navigation Data and justified ActiveNavigation pixel calculus values exceed a tolerance threshold) containedin the current image are processed by the Neural Network. The benefitsof this can be illustrated by taking the hazardous condition example ofa car in the impact zone. FIG. 5 is a representation of the image fromwhich conventional computer vision would need to declare and classify ahazardous condition due to the presence of a car 502 near the track.Comparatively the present system has already declared a hazardouscondition from Active Navigation's out-of-tolerance pixel detectionwithin the predefined impact zone, and those out of tolerance pixels asrepresented in FIG. 6 is all that's required for the Neural Network toclassify that hazardous condition as an automobile. Active Navigationtreats these out of tolerance pixels as a pixel signature, provided thepixel signature remains within predetermined limits for both pixelvariance and location. In this manner, a hazardous condition classifiedas a car need not be re-submitted to the Neural Network for objectclassification on each successive image (record) processed by ActiveNavigation. Instead, a temporary list of pixel signatures is maintainedfrom image to image until the train has passed the location of the pixelsignature in order to recognize that object classification for the pixelsignature need not be performed again. However, if a change occurs, suchas a human entering the impact zone at a point near the car, either orboth of the predetermined limits for pixel variance and location wouldbe exceeded, which would trigger a Neural Network classification event.The pixel signature provides a unique reference for the tracking andhazardous condition response by the present system's rail computervision components, Active Navigation, Neural Network, Object Trackingand Train Handling and Hazard Response.

Continuing with a high-level overview, consider the third component ofthe present system's computer vision, Object Tracking. The ObjectTracking processor of the onboard computer vision system consumes pixelsignatures from Active Navigation and if available, the hazardclassification from the Neural Network. Object Tracking monitorshazardous conditions to determine if they are moving in or out of theimpact zone or failing to move at all. Using Baseline Navigational Data,grids are applied to the trapezoids in the active view to track a pixelsignature within the trapezoidal framework. Grids may vary in size tooptimize Object Tracking at different distances from the locomotive.Object Tracking can then determine if hazardous conditions areresponding to locomotive warning devices by referencing movement againsttrapezoidal grid boxes. This trapezoidal referencing is depicted in FIG.7A-7B. Initially the SUV 702 only occupies the grid boxes on the rightside of the trapezoid, however in the next sampling the SUV 702 occupiesgrid boxes on the right, center and left side of the trapezoid allowingthe present system in infer movement of the SUV from right to left asshown in FIG. 7B. Noted too in FIG. 7B, the locomotive is now closer tothe SUV 702 and the reference trapezoid is proportionately largerhowever its orientation to the track is consistent, allowing the presentsystem to monitor the SUV's progression across the track. Should the SUVspan trapezoidal boundaries on subsequent image captures, the presentsystem still ascertains the same number of out of tolerance grid boxesoriented to right, center or left of the trapezoidal reference.Utilizing trapezoidal reference in this manner extends to zoomed imagesas well in cases where a hazardous condition is detected in trapezoidsat greater distances from the locomotive and the increased pixel countfrom a zoomed images can be deterministic. In this way, Object Trackingfeature is similar to Active Navigation and Neural Network in that itdirectly benefits from the trapezoidal position information of BaselineNavigational Data. Conventional computer vision typically employs pixelimage subtraction methods to determine if an object has moved betweentwo images. Image subtraction becomes more onerous when both the objectand locomotive are moving. Conversely, Object Tracking is constantlyreceiving pixel signature updates from Active Navigation within theBaseline Navigational Data framework. Consequently, the pixel signaturecan be tracked within the trapezoidal grid subdivisions of the activeview. In FIG. 7A-7B for example, a vehicle's progression crossing thetracks can be monitored by the movement of the out of tolerance pixelsthrough the trapezoid which is accomplished within the trapezoidalframework provided by Baseline Navigational Data and without thecomputational and interpretive burdens presented by image subtraction.

Object Tracking predicts hazardous condition states for Train Handlingand Hazard Response facilitating responses to hazardous conditionsentering or leaving the impact zone. Train Handling and Hazard Responseincorporates information from Object Tracking combined with data on thecurrent operating characteristics of the train, including stoppingdistances for both penalty and emergency brake applications to meet orexceed Industry best-practices for hazardous condition response.

Train Handling and Hazard Response monitors Active Navigation, ObjectTracking and Neural Network as well as current braking distance asfurnished by PTC. Based upon information gather during the developmentphase when the system is logging locomotive engineer's responses todynamic hazardous conditions and input from railroad operation'spractices personnel, use cases are developed which determine TrainHandling and Hazard Response's adjudication for when to apply warningdevices, throttle down or train brake. Train Handling and HazardResponse interfaces with EM, PTC, PTL, onboard or wayside drones,warning devices and the dispatch center to evaluate and commandresponses to detected hazardous conditions.

A more in-depth examination of the processes in FIG. 2 begins with theBaseline Navigational Data creation process which is the basis for thedisclosed invention's unique approach to computer vision. When deployingthis system, unlike conventional computer vision systems, railroads mustmake a significant investment in the creation and maintenance ofBaseline Navigational Data. However, as will further be demonstrated,this endeavor will optimize hazardous condition identification,classification, and object tracking as well as Train Handling and HazardResponse.

Baseline Navigational Data

Unlike highway vehicles, rail trips between rail terminals are highlyrepetitive, occurring on line segments referred to as districts orsubdivisions. These well-defined trips lend themselves to the collectionand development of Baseline Navigational Data, which can then bereferenced by Active Navigation to greatly reduce background and fixedobject noise resulting in more robust hazardous condition detection andclassification. Once developed, Baseline Navigational Data is assembledby the disclosed system's processes into a file which is downloaded bythe locomotive during or before a system initialization process prior tobeginning an automated trip.

The development of Baseline Navigational Data begins in the field withtraversing a data-collection locomotive over the candidate subdivision,equipped with the disclosed system's hardware running itsdata-collection software which interfaces with the onboard PTL system.PTL is comprised of a GPS receiver, a ground-based correction signalreceiver, wheel tachometer input and an inertial measurement unit (IMU).PTL provides sub-foot location accuracy which is leveraged by thepresent system for both Baseline Navigation Data creation and ActiveNavigation.

Onboard the data-collection locomotive, the disclosed system's softwareapplies precision location information to captured images andunobstructed distance values, storing them at sample rates which supportthe post-trip development requirements for Baseline Navigational Data.The disclosed system's onboard Baseline Navigational Data developmentsoftware uses inputs from PTL to collect and annotate images for theBaseline Navigational Data development process. In one exemplaryembodiment of the disclosed system's Baseline Navigational Data creationprocess, precise location attribution will be provided for every 11 feetof locomotive travel along the track of the subdivision. For a traintraveling 60 mph (88 ft/sec), this would provide Active Navigation areference in Baseline Navigational Data every 0.125 seconds.

In the post-trip development process, images and data from thedata-collection locomotive are converted to sets of numerical values forthe Baseline Navigational Data file. When used by Active Navigation,these values provide the roadmap for where to “look”, i.e. (x,y)coordinates (FIG. 3 ), and what to “expect”, in terms of pixel calculusand unobstructed distance values, when no hazardous conditions arepresent. Baseline Navigational Data will be developed for allsubdivisions for which automation is desired and may be organized suchthat each direction of travel is its own Baseline Navigational Datafile. All subdivision multiple-track configurations and sidings whichcould be traversed in automated operations will require BaselineNavigational Data.

The present system's Baseline Navigational Data auto-creation processbegins by navigating imagery collected from the data-collectionlocomotive. The lab version of the present system's Neural Network isutilized in this process. The lab Neural Network is similar to ActiveNavigation's Neural Network absent the time constraints of the activesystem, as such more neural layering is possible for higher accuracyduring the data development process. The lab Neural Network identifiesthe rail being traversed and any hazardous condition present during thecollection process.

Accurate identification of the rail being traversed, which may becomplicated by multi-track locations, is essential for the definition ofimpact and warning zone coordinates. As such the Baseline NavigationalData development tools are able to navigate forward and backward throughthe collected data to maintain route integrity throughout the creationprocess. Allowing for accurate assignment of (x,y) coordinates for theimpact and warning zones irrespective of multi-track configurations.Similarly this process aids in the establishment of the active viewalignment information for inclusion in Baseline Navigational Data whichallows the Active Navigation to apply Baseline Navigational Data to theactive view without errors from vibration introduced by a movinglocomotive.

The final application of the lab Neural Network is for identification ofhazardous conditions in the impact and warning zones which were presentduring the data-collection process. For instance, consider a vehicle ata highway grade crossing passing in front of the data-collectionlocomotive. The auto-creation software tools will scrub the pixels inthe impact and warning zones of any hazardous conditions before creationof the Baseline Navigational Data file by substituting pixels fromadjacent imagery frames or substituting pixels from the surrounds of theaffected frame. Similarly, these same tools will identify fixed objectsin the warning zones which can be flagged in the Baseline NavigationalData file for special handling so that onboard classification resourcesare conserved. By this methodology the auto-creation processes in thelab orients trapezoids to defined areas of concern around the track andidentifies fixed objects as non-hazardous conditions, greatly reducingcomputer vision resource demands.

After the field collected images have been scrubbed of any hazardousconditions, the disclosed system's lab software tools convert storedimages into sets of numerical values for the Baseline Navigational Datafile. When used by Active Navigation, again these values provide theroadmap for where to “look”, i.e. (x,y) coordinates for the impact andwarning zones and what to “expect”, in terms of pixel calculus if nohazardous conditions are present. Baseline Navigational Data will needto be developed for all autonomous subdivisions and may be organizedsuch that each direction of travel is its own Baseline Navigational Datafile. All multi-track and sidings which could be traversed in autonomousoperations would also require Baseline Navigational Data which again maybe organized as individual files for each direction of travel.

The preferred methodology for the Baseline Navigational Dataauto-creation begins by processing the imagery collected by the datacollection device (e.g., a capture device on a locomotive) during a datacollection trip in the reverse direction, starting with defining thecoordinates of, and calculating the pixel calculus value (or values ifmultiple grid boxes are used) for, the final impact zone trapezoid andwarning zone trapezoids corresponding to a location at the end of thedata collection trip, and ending with defining the coordinates for, andcalculating the pixel calculus values(s) for, the first impact zonetrapezoid and warning zone trapezoids corresponding to a location at thestart of the data collection trip.

An exemplary impact zone trapezoid 802 is depicted in FIG. 8 (also shownin FIG. 8 are warning zones 804, 806 on either side of the impact zone802). As shown in FIG. 8 , the impact zone trapezoid 802 is defined byfour pairs of coordinates (x1,y1; x2,y2; x3,y3; and x4,y4). A NeuralNetwork focused solely on identifying the rail is used in addition topredetermined measured pixel counts representing exemplary 11 ft of(1-n) linear track distance for a chosen image capture device ofprescribed resolution. By this methodology, the pixel count for examplebetween (x1,y1) and (x3,y3) is known for a given image capture device,similarly the pixel count between (x1,y1) and (x3,y3) of this trapezoidcan be predetermined for (1-n) occurrences of this trapezoid in view ofthe reference locomotive. The pixel counts between exemplary lateralcoordinates (x3,y3) and (x4,y4) follow the same methodology ofpredetermined measured pixel counts for (1-n) increments from thereference locomotive. Thus, creation of baseline navigational data forthe impact and warning zones consists of the Neural Network identifyingthe rail followed by application of predetermined pixel counts to definetrapezoid dimensions. In FIG. 9 , the iterative process continues byworking backward to process an image taken a location corresponding tothe penultimate trapezoid 904 of the subdivision. At this location, theNeural Network again first identifies the rail and then the (x,y)trapezoid coordinates as before. The (x3,y3) and (x4,y4) coordinates ofthe newly defined penultimate trapezoid 1004 are shared points with thetrapezoid 802 defined in FIG. 8 (reference numeral 1002 in FIG. 10 ),which effectively appends the two impact zone trapezoids 1002, 1004together as depicted in FIG. 10 . Additionally, the size of the finalimpact zone trapezoid 1002 as depicted in FIG. 10 is now decreased by alearned pixel count to account for its increased distance from the datacapture device.

This methodology of appending previously defined trapezoids in priorimages to the current images creates frame coupling between distinctimages and provides the trapezoidal reference data necessary for ActiveNavigation to process its active view.

This process continues by iterating backward until the first impact zonetrapezoid in the subdivision is reached. FIG. 17 further illustrates thefirst five steps of this process in one embodiment for the impact zonetrapezoids (the process for the warning zone trapezoids is similar). Byway of example, if the data-collection locomotive traversed thesubdivision from south to north, in Step A as shown in FIG. 17A thenorthern-most (final) impact zone trapezoid 1702 is defined first. Thisdefinition includes trapezoidal (x,y) coordinates and pixel calculus(which again may be for the entire trapezoid or for multiple grids inthe trapezoid). In Step B in FIG. 17B, the reference locomotive is nowone trapezoid length south of its Step A position and the next(penultimate) impact zone trapezoid 1704 is defined. As shown in FIG.17C at Step C, the trapezoid 1702 from 17A is appended to the newlydefined trapezoid 1704 in 17B and sized to coincide with its newposition relative to the reference locomotive. New pixel calculusvalue(s) for both the final impact zone trapezoid 1702 (because itsposition is now father from the reference locomotive and its dimensionshave changed) and for the penultimate impact zone trapezoid 1704 (again,multiple pixel calculus values for each grid in a trapezoid may becalculated for embodiments that employ grids) are determined. In Step Din FIG. 17D, the process continues with the definition of athird-to-last trapezoid 1706 immediately in front of the locomotive. InStep E of FIG. 17 e , the final and penultimate two trapezoids 1702,1704 are appended to the newly defined third-to-last trapezoid 1706 fromStep D. Again, new pixel calculus values are determined for all threetrapezoids. By this methodology each time a trapezoid is originallydefined it is immediately in front of the locomotive and then continuesto be redefined with each iterative step until it is no longer in view.A typical subdivision defined by this methodology may contain a recordrepresenting every 11 feet of linear track containing the data for eachtrapezoid in view of the “reference” data collection locomotive.

Each time a trapezoid is defined, so too is its pixel calculus. Thefinal trapezoid 802 as defined in FIG. 8 has a pixel calculus which isthen redefined with a new pixel calculus calculation in FIG. 9 , whereit is now the second trapezoid 902 from the reference locomotive. Thisiterative process continues for each trapezoid until it is no longer inview of the reference locomotive.

As discussed above, in some embodiments, coordinates are determined andpixel calculus values are calculated for grid boxes superimposed on thetrapezoids. Grid boxes vary in size and number depending on atrapezoid's distance from the reference locomotive with trapezoidscloser to the locomotive having larger numbers of grids than trapezoidslocated further from the locomotive. FIG. 11 is a depiction of a gridbox within a trapezoidal definition, a bit weighting (of 1, 2, or 3) isapplied to lessening the effects of vibration and any misalignment ofBaseline Navigational Data and the active view. In one embodiment of thepresent disclosure, grid boxes located on a side of the trapezoid willhave a shape with a side that aligns with the side of the trapezoid,with the other side being vertical and aligned with the neighboringsquare-shaped grid box, and horizontal tops and bottoms aligned with thehorizontal top or bottom of the trapezoid and/or grid boxes locatedabove or below as depicted in FIG. 19 .

The Baseline Navigational Data file can also contain pixel calculus fordifferent times of day and, in some embodiments, even different times ofyear, and/or weather conditions (e.g., rain, snow). Rail's highlyrepetitive routes lend themselves to acquiring extensive data on thesesubdivisions and the disclosed system's lab and development processesallows every train that traverses a subdivision to validate andpotentially drive data improvements for the subdivision.

Despite the efforts listed above, pixel calculus can be affected bydifferences between conditions during the collection process and currentoperating conditions. Preferred embodiments of the disclosed systemaddress this disparity with a justification process that utilizesjustification windows. The justification process allows the activesystem to conserve its tolerance budget and reliably identify hazardousconditions in varying operating conditions. Baseline Navigational Datawill contain definitions for one or more justification windows. In someembodiments, a single justification window is used. In otherembodiments, multiple justification windows, e.g., three justificationwindows, are used. In some embodiments, a justification windowcorresponds to a 20×20 pixel window. In preferred embodiments, a pixelweighting such as that depicted in FIG. 11 is applied to the pixelvalues in the justification window. The weighted pixel summation for thepixels in the window, or the average weighted pixel summation for eachof multiple justification windows, is divided by the total number ofpixels in the justification window (s) and is stored as a baselinejustification value, in the Baseline Navigational Data file for eachrecord. When active view data is received, justification window data iscomputed for the justification window(s) corresponding to thecoordinates in the Baseline Navigation Data, including the applicationof the weighting scheme (if any) reflected in the Baseline NavigationData, and used to compute an active view justification value. Thedifference between the baseline and active view justification values isthen calculated and applied as a justification factor to each pixel inthe active view data prior to pixel calculus being performed on theactive view data. Additionally, the difference between the baseline andactive view justification values is then calculated and divided by thenumber of pixels in the justification window (or windows if there aremore than one) to arrive at a per-pixel value which is applied to eachpixel in the active view data prior to pixel calculus being performed onthe active view data.

In one exemplary embodiment the justification window(s) designation(s)would be in the leading edge of the impact zone immediately in front ofthe reference locomotive for Baseline Navigation Data. In this manner,any hazardous condition would already have been detected before being inthis proximity to an active locomotive. The justification window(s)coordinates are preferably fixed relative to a capture source and storedin the system's database. However, it is possible to employ variablejustification windows, in which case the coordinates would be stored inthe corresponding record in the Baseline Navigation Data. This allowsfor a valid comparison between Baseline Navigational Data and the activeview justification windows with no hazardous conditions present. Theresult of this comparison is used to adjust (justify) the pixel valuesin the impact and warning zones of the active view, so that thejustified active view may be now compared to Baseline Navigational Datawithout distortion caused by environmental differences between currentconditions and conditions at the time the data used to derive theBaseline Navigation Data was collected.

The Baseline Navigational Data development tools use PTL precision GPScoordinates collected in the field and railroad data to interpret trackcurvature. Employing a rail-trained Neural Network in the lab, the toolsidentify the rail and create the trapezoidal definitions, orienting themcorrectly to the track in accordance with its rail-trained computervision and its understanding of curvature from railroad data or bearinginterpreted from precision GPS readings. Exemplary FIG. 12 is anillustration of impact zones in a 2.5 degree curve in one embodiment.

An embodiment of the disclosed system may organize the BaselineNavigational Data file by precise location such that all (x,y) trapezoidcoordinate information in the active view would represent a singlerecord for that location within the Baseline Navigational Data file. Inthe FIG. 12 example using this methodology, 9 trapezoids would becontained in this Baseline Navigational Data record for the location.Other records may contain many more trapezoids as dictated by theavailable sight distance.

The size and scope of the development of the Baseline Navigational Datausing the disclosed systems tools is significant. The simplification ofhazardous condition detection by using Baseline Navigational Data andits offline computational data comes with considerable effort. Forperspective, the auto-creation process applied over a typical 200-milesubdivision would generate a Baseline Navigational Data file containingover 90,000 location records and upwards of 2 million trapezoiddefinitions in some embodiments.

Table 1 below is an exemplary representation of a Baseline NavigationalData record. The example in Table 1 assumes that the justificationwindow coordinates are fixed relative to a capture source, so that it isnot necessary to store justification window coordinates in each record.In alternate embodiments, each record also includes (x,y) coordinatesfor one or more justification windows.

TABLE 2 high level description of an exemplary Baseline NavigationalData Record Attribute Comment Precision Location of required for ActiveNavigation sync to Capture Device Baseline Navigational Data fileTrapezoidal (X, Y) Defines specific areas of interest for Coordinatesonboard resources: Impact zone Warning zone Defines/optimizestrapezoidal grid size Organized by source: Normal locomotive view Zoomedlocomotive view Drone view Fixed wayside device view Pixel CalculusPixel summations for each trapezoid (impact zone and warning zones) orfor each grid box in each trapezoid for a particular source/view, andfor a particular time of day and/or time of year. Pixel calculus valuesare used for: Hazard detection Classification Object tracking Supports:Varying tolerance values (the grid sizes within a trapezoid areconsistent, but grid sizes in a trapezoid further ahead of the train arelarger than the grid sizes in a trapezoid closer to the train) AlignmentReference for One or more pixel coordinate values Active Viewcorresponding to one or both rails Unobstructed Distance Reflectivetechnology measured distance Capture Device Controls Zoom Pan DroneFlight Options Static drone coverage areas Available drone flightpatterns Grade Crossing Attribution Design start time in advance oftrain arrival Interlocked with traffic signals Gate and lightmalfunction detection Multi-Track and Fixed Location where equipment onadjacent tracks Object Attribution and fixed objects affect sightdistance Platform, Bridges, Tunnel Predetermined warning areas LocationsPredetermined train handling factor areas Baseline Justification Valueof weighted pixel summation for a Window Value justification window oraverage of weighted pixel summations for multiple justification windows,as applicable, divided by the number of pixels in the justificationwindow(s) used for justifying corresponding active view data

Detailed Description of Baseline Navigational Data Attributes

Precision Location of Capture Device—an attribute in BaselineNavigational Data which allows Active Navigation to access the correctrecord from the file for its hazard detection analysis. The presentsystem requires precise location information for both the BaselineNavigational Data file and for Active Navigation as will be described indetail in the Detailed Description of the Production System section ofthis text.

Trapezoidal (X,Y) Coordinates—attribute in the Baseline NavigationalData which when applied to Active Navigation defines areas of interestaround the track, i.e., impact zone and warning zone; also defined isgrid sizing for trapezoids in the impact and warning zones. Trapezoidal(X,Y) coordinates may be organized by capture source, i.e. locomotive,drone, and fixed wayside. While the impact zone remains consistent inits relationship to the track, for the warning zone this can vary.Instances where steep embankments make access to the track impossible,the warning zone could be very small. Conversely, at grade crossings thewarning zone may expand to detect approaching vehicles and properoperation of the crossing gates and lights, an activity performed byonboard train personnel today. In one embodiment of the presentdisclosure a (z) coordinate may be defined. For example, the z-axis maybe useful in defining the elevation of the top of the rail whichprovides a useful clearance reference for Active Navigation when minordebris is in the track or when operating with snow on the ground. ABaseline Navigational Data file may contain separate (x,y) coordinateinformation for different input devices or input device settings,including the normal locomotive image capture device(s), a zoomed imagefrom the locomotive capture device, images from drones or deployed fixedwayside devices. An embodiment of the disclosed system may use the sameBaseline Navigational Data for both the normal locomotive view and adrone with a similar capture device whose collection flight mimics theorientation of the locomotive capture device.

Pixel calculus—an attribute in the Baseline Navigational Data ofweighted pixel summation performed on individual pixels or aggregationsof pixels for each trapezoid and/or trapezoidal grid box. In someembodiments of the present disclosure a grid may be superimposed over atrapezoid, that grid may vary in size, such that trapezoids closer tothe locomotive have a denser grid pattern than those farther away. Whena grid is applied a separate pixel calculus will be established for eachgrid box, providing more localized hazardous condition detection,classification and object tracking. Tolerance levels are associated witheach grid box and may vary with distance from the acquisition device.Tolerance levels may also be independently applied by Active Navigationto adjust for environmental factors.

Alignment Reference for Active view—attribute containing one or morepixel coordinates for the active rail which may be applied to the activeview in order the align the active view with Baseline Navigational Datafor computational purposes.

Unobstructed distance—attribute collected by reflective technology atstrategic locations as prescribed by design criteria and identified byprecision GPS coordinates in Baseline Navigational Data, comprising adistance measurement in linear track feet for which no obstructionshould be present. Active Navigation can compare this value to currentvalues to determine the presence of a hazardous condition. FIG. 4depicts the use of this technology which typically would occur when alocomotive encounters longer sections of tangent track.

Capture Device Controls—attribute may be either static or dynamic.Static controls are established during the present system's locomotivedata-collection process. Strategic zoom images are captured during thisprocess at prescribed criteria for available sight distance. In someembodiments pan and tilt controls could also be defined. However, thesystem's reliability and availability may benefit from image capturedevices having fewer moving components, such as a second capture devicescapable of a wide-angle view. Dynamic zoom control, while not anavigational database attribute, is also enhanced by the presentsystem's Baseline Navigational Data. A detected hazardous conditiontriggering a dynamic zoom event would utilize the inherent trapezoidalranging from an out of tolerance trapezoid to affect proper zoomcontrols. As will be further explored, in the hazardous conditiondetection and response portion of this text, trapezoids provide animmediate reference for distance between the locomotive and thehazardous condition.

The drone flight option attribute designates predefined areas for dronecoverage and indicates available drone flight patterns for use by ActiveNavigation. This attribute would also restrict drone usage in prohibitedareas. Drones may be dispatched from locomotives or wayside stations.

The grade crossing attribute contains pertinent data for each gradecrossing on the automated route. This includes but is not limited to:configuration of installed warning devices (allows the present system toverify proper operation of gates and light as currently performed bytrain crew), design start time in advance of train's arrival at thecrossing and if warning device is interfaced with highway trafficsignals designed to move vehicles off the track for an approaching train

The multi-track attribute designates track that may have the presentsystem's evaluation distance reduced when equipment is on an adjacenttrack. Based upon curvature of the two tracks, the navigational databasecan so designate these areas. Dispatch center updates and/or presentsystem detection and classification processes can identify equipment onadjacent tracks triggering the present system to operate with reducedevaluation and response distances. FIG. 18 is an example of reducedimpact and warning zone evaluation and response distances resulting fromequipment on an adjacent track. Similarly, the fixed object attributeidentifies permanent obstructions such as bridge abutments, propanetanks, and wayside signal structures which can be flagged asnon-hazardous conditions thus conserving onboard computer visionresources.

The platform, bridges and tunnels attribute designate areas wherelocomotive engineers typically provide warning of the approaching train.These designations in the Baseline Navigational Data allow for anauto-generated sounding of the train horn or bell at a prescribedapproach distances.

Detailed Description of a System According to an Embodiment

The present disclosure provides generally for a system, methodology,processes, and apparatus for automated response to line-of-sighthazardous conditions. Active Navigation, Neural Network, Object Trackingand Train Handling and Hazard Response comprise the present system'scomputer vision software, processes and methodologies. According to thepresent disclosure, enhanced hazard-detection techniques are used inconnection with existing speed controls to better predict and preventcollisions or other undesirable events by regulating the speed of thetrain.

Active Navigation and Synchronization with Baseline Navigational Data

Autonomous train operations begin by ensuring Baseline Navigational Datais downloaded to the locomotive before trip commencement. Systeminitialization will be executed between the lead locomotive hosting thepresent systems computer vision system and the railroad back office ordispatch center. Among other checks, the initialization will confirmsystem health, proper communication between the present system and bothPTC and EM, the automated route and download of the correct BaselineNavigational Data file for the intended trip (if not already storedonboard). All possible candidate Baseline Navigational Data files,including multiple mainline tracks and sidings, are downloaded from theback office or dispatch center for all subdivisions in the planned trip.Active Navigation would assemble these Baseline Navigational Data filesinto its active memory so that while the train is operating, any updatesfrom PTC regarding track routing, are seamless.

Once positioned at the entry point of an autonomous subdivision, thelocomotive can begin the trip with Energy Management controlling thethrottle and routine brake applications, while PTC controls the penaltybrake and emergency brake applications required to stop a train withinprescribed stopping distances. The disclosed system interacting withthese two systems provides responses to hazardous conditions in andaround the track, a function previously performed by the locomotiveengineer.

Referencing FIG. 2 , at the trip's commencement, Active Navigationbecomes located by obtaining its present location from the PTL input.Once located, Active Navigation references the Baseline NavigationalData file record which corresponds to its current location. Onceunderway, Active Navigation navigates using PTL and anticipates the nextcorresponding location record in the Baseline Navigational Data file ofconsecutive location records. Active Navigation then captures the nextcorresponding image from the Locomotive Image Capture Device foranalysis. In this manner, by triggering on the next available BaselineNavigational Data record, Active Navigation is able to synchronize itsactive view with the Baseline Navigational Data file.

Active Navigation can synchronize multiple active views simultaneouslywith Baseline Navigational Data. In addition to the normal image fromLocomotive Image Capture Device, a zoomed image from this device ordedicated device can be synchronized with Baseline Navigational Data.Active Navigation's monitoring of the Baseline Navigational Data fileallows Active Navigation to send zoom controls to the Locomotive ImageCapture Device and trigger an image capture to correspond to thesepredefined locations in the Baseline Navigational Data file. Areas oftrack where a locomotive coming out of a curve gains a 2500 foot view ofthe intended route are good examples of statically zoomed data containedwithin the Baseline Navigational Data file. Additionally, ActiveNavigation may send dynamic zoom commands to the Locomotive ImageCapture Device(s) if an out of tolerance pixel calculus is detected. Thetrapezoid containing the pixel signature provides Active Navigation thetrapezoidal range for affecting accurate zoom controls to the LocomotiveImage Capture Device(s) potentially allowing the disclosed system tomake a hazardous condition detection and classification from a zoomedimage.

A Remote Drone Capture Device may provide another simultaneous viewwhich can be supported by Active Navigation. Drones can be flown with aflight pattern that mimics the rail orientation of the Locomotive ImageCapture Device allowing them to use the same Baseline Navigational Dataprovided proper compatibility between the two capture devices. Inembodiments of the present disclosure where this is not the case,separate drone Baseline Navigational Data can be developed in a similarprocess as previously presented. Drones are not required to implementthe disclosed system but when integrated into the present system's rapidscreening processes (of where to “look” and what to “see”) to detecthazardous conditions against Baseline Navigational Data, drones have thepotential to further increase safety of operations for both theirability to provide warning to the public of an approaching train anddetection of hazardous conditions at distances which allow ample time tobring the train to a safe stop.

Remote Drone Capture Device usage may be static or dynamic. Static usageis defined as areas in the Baseline Navigational Data file where dronesare always dispatched. These may include areas of limited sightdistance, areas prone to rockslides or washouts, tunnels or gradecrossings. Weather permitting, static drone usage would always occurduring automated operation for the prescribed locations. The disclosedsystem can calculate when to dispatch the drone in advance of the areadesignated for static drone coverage by calculating the time requiredfor the drone to arrive at the start of the desired coverage area inadvance of the train's PTC calculated stopping distance.

Dynamic usage of Remote Drone Capture Device is defined as a response toan unexpected event. This might be caused by encountering a PTC mandatedRestricted Speed target. In this case, a drone would be dispatched bythe present system to determine what may have caused a more restrictivesignal indication and augment compliance with the Restricted Speedmandate to operate at a speed capable of stopping within one-half therange of vision. Additional examples of dynamic drone usage includetrain inspection after an emergency braking event or a wayside equipmentdetector alarm calling for train inspection.

Hazardous Condition Identification, Classification and Object Tracking

The disclosed system's transformative approach to computer vision whichleverages Baseline Navigational Data as a no hazardous condition presenttruth from which Active Navigation can declare a hazardous condition ina manner which conserves computer resources and improves computer visionfunctionality for classifying hazardous conditions in less time. Abetter understanding of the advantages of the present system can betaken from a comparison with conventional computer vision. Whileconventional computer vision systems are far from monolithic andever-evolving, they do contain common components. Convolutional NeuralNetworks are considered necessary for the most advanced computer visionsystems currently. These consist of convolutional layers which identifyobjects and classify them. Typically, this involves sliding different3×3 filters over all the image pixels. These filters expose imagefeatures which when processed by the e.g., 24 convolutional layers, theoutput of which can then classify the object in what is termed the“connected layer” where a voting process yields a final classificationwith a degree of certainty at which point for rail computer visionpurposes, a hazardous condition could be declared. Comparatively, thedisclosed system declares a hazardous condition in fewer less complexsteps, by comparing Active Navigation's active view pixel calculus andunobstructed distance values e with those in Baseline Navigational Data.Both active view pixel calculus and unobstructed distance arestraight-forward computation that can easily be returned in far lessthan 10 msec. Once Active Navigation declares a hazardous condition, theNeural Network can add classification context (e.g., deer, person, car).If the Neural Network fails to identify the hazardous condition withacceptable certainty or within an acceptable time period as determinedby Train Handling and Hazard Response, it will be treated in a fail safemanner by assigning the highest severity index to the hazardouscondition. Transforming computer vision in this way makes sense for railoperations in that the immediate declaration of a hazardous conditionsbefore classification is a conservative and prudent approach appropriatefor the physics of stopping a large freight train.

The Neural Network as has been established only consumes the pixelsignatures as identified by Active Navigation. Performing classificationon this limited subset of pixels has several advantages. The first beingNeural Networks with more layers tend to produce better results. Morelayers become a possibility with the present system because less time isrequired for the system's software to evaluate the smaller data set.Secondly because the pixel signatures are associated with a giventrapezoid, a scaling factor can be applied to the classificationprocess. The present system implicitly knows the distance each trapezoidis in advance of the train. This makes the present system'sclassification more precise in that it inherently knows if it'sidentifying a person in the track at 200 feet or 2000 feet. Thedisclosed system's Neural Network will be developed and trainedspecifically for rail operations. Image characteristics needed to traina rail specific Neural Network are primarily focused on human beings,highway vehicles, animals, railcars, locomotives, trees, poles, rockslides, washouts, and misaligned track. The design of the disclosedsystem facilitates a Neural Network which concentrates onboard resourceson classifying the most common hazardous conditions sooner and moreaccurately than conventional computer vision systems. This novel systemarchitecture supports this by allowing Active Navigation to declare ahazardous condition which immediately imposes a pending braking eventunless the Neural Network can classify the hazardous condition within atime period comparable to a locomotive engineer's recognition andreaction time for applying the brakes manually. An example of thismethodology begins with Active Navigation detecting out of tolerancepixels and declaring a hazardous condition. If the Neural Networkdetermines this to be a deer in the impact zone, then the system wouldmerely sound the warning device. However failure of the Neural Networkto classify the hazardous condition in the prescribed time results intreating the unclassified hazardous condition as a highest severityevent. This fail-safe approach to hazardous condition classificationmarkedly distinguishes the disclosed system from conventional computervision which relies on Neural Networks without the level of screeningprovided by highly developed Baseline Navigational Data. Furtherdistinction of the present system's approach to hazardous conditionclassification is that Neural Network need not be trained to recognizerandom items such as sink holes, downed radio towers or even airplanesthat may present as hazardous conditions. Active Navigation will declarethese one-off events as hazardous conditions based on out of tolerancepixels, making it inconsequential that the present system's NeuralNetwork cannot classify them. This approach focuses the present system'sNeural Network resources on the precise and timely classifications ofhazardous conditions which occur more frequently and bearinterpretation, such as a vehicle on the track with its hood raised or alowboy trailer stuck on the tracks.

Object Tracking benefits from out of tolerance pixels screening in asimilar fashion as Neural Network. A hazardous condition identified byActive Navigation can be tracked by calculating the movement of thepixel signature. Active Navigation can coincidentally establish multiplepixel signatures to accommodate different out of tolerance pixelclusters. Object Tracking, Neural Network, Active Navigation and TrainHandling and Hazard Response can now distinguish between differentsimultaneous hazardous conditions and communicate effectively, providingupdates on the individually identified pixel signatures. Object Trackingcan associate updates from Active Navigation with the pixel signatureand compare the position of the pixel signature with that from theprevious update. In this manner, Object Tracking can then makepredictions based on the movement of the hazardous condition as to whenit would be clear of the impact zone, much the way the locomotiveengineer does today. In cases where pixel signatures are outside theimpact zone, Object Tracking may predict when they would enter theimpact zone. Additionally, Object Tracking may determine if a hazardouscondition is responding to locomotive warning devices. AccomplishingObject Tracking against the backdrop of Baseline Navigational Databecomes straight forward in that movement within the trapezoidalreference is readily apparent. Comparatively, conventional computervision would process consecutive image updates and subtract one from theother to determine position change of the object in question. This iscomplicated further by the fact that the train is moving. As such,conventional computer vision would need to identify a fixed object toreference in multiple images to perform image subtraction for a movingtrain. Again, FIG. 7 is a representation of the effect BaselineNavigational Data has on Object Tracking. Baseline Navigational Data'strapezoid provides an immediate reference irrespective of how fast thetrain is traveling.

Train Handling and Hazard Response

Despite impressive computer vision systems for highway navigation, mostif not all are still in the realm of driver assist systems. Rail poseseven greater challenges for autonomous operations, given freight trainstopping distances routinely exceed 5000 feet and as a practical matter,trains because of their braking physics and available sight distances,may not be able to stop short of a potentially hazardous conditions evenwhen immediately detected by human operators or computer vision systems.Compounding the challenges of rail operations is the unreasonableness ofimmediately placing a train in an emergency braking application becausea hazard, even a person, is detected in the track 2500 feet in advanceof a train. This would be an untenable practice for rail operations andone that locomotive engineers routinely encounter without invoking anemergency braking response. Another common example of these type ofintense events which locomotive engineers typically must endure (“rideout”), occurs at highway grade crossings. When equipped with warningdevices, highway grade crossings are typically designed to provide 30seconds of warning to a motorist before the arrival of a train. However,a train traveling 30 mph would typically not activate the warning deviceuntil it was approximately 1400 feet from the crossing. Depending ontrack grade and train consist, this could result in a vehicle stoppedmomentarily on the tracks at less than the train's stopping distancebefore the motorist is warned of an approaching train by the highwaygrade crossing warning device. Again, initiating an emergency brakeapplication in response to this common situation would be impracticaldue to the dramatic nature of an emergency brake application and thereal prospects of creating exposure to derailment multiple times onevery trip. A better response mimics the locomotive engineer whounderstands when the crossing signal activates and when to expect anyvehicles to be clear of the impact zone.

Unquestionably, the vast majority of locomotive engineers do anexceptional job of responding to incursions into the path of the train,but as with any human behaviors, there is a wide spectrum ofstimulus/response which may depend on a given engineer's skills,experience, alertness or simply his/her perception at any given moment.What is needed is a system that responds to these situations consistentwith locomotive engineer's best operating practices with the increasedvigilance and the consistency of a computer vision system. A systemwhich optimizes early detection coupled with appropriate warning andbraking events will improve safety of operations for the rail network.The disclosed system's development process would include comparing thehazardous condition responses of the locomotive engineer to that of thepresent system. During the development phase the present system wouldlog its response to different hazardous condition events, whilesimultaneously logging the actions taken by the locomotive engineerstill in full control of the train. These activities would serve torefine system use cases.

Train Handling and Hazard Response in one embodiment of the presentsystem, can assign a severity index (or risk profile) to a variety ofhazardous conditions detected by Active Navigation. Hazardous conditionsare represented by a distinct pixel signature number which allows theevent to be updated by Active Navigation, Neural Network and ObjectTracking. The updates provided to Train Handling and Hazard Response mayalter the severity index and hence the response. At a minimum, updatesoccur each time Active Navigation access the next Baseline NavigationalData record and performs pixel calculus which will continue to identifythe hazardous condition by its pixel signature, provided the signatureremains within prescribed limits. The present system POC demonstrateshow the magnitude of out of tolerance pixels provides a pixel signaturein the current active view that is identifiable by Active Navigation,allowing the hazardous condition to be updated and tracked when comparedto the previous active view.

The Neural Network and Object Tracking provide classification andtracking updates respectively to Train Handling and Hazard Response.These updates are used by Train Handling and Hazard Response tocontextualize hazardous conditions. Highway grade crossings is an areawhere the present system's specific rail-trained Neural Network issignificant. While incidents of a vehicle on or near the track of anapproaching train will always occur, Train Handling and Hazard Responsewould immediately change its normal response to this situation if NeuralNetwork detected the hood of the vehicle raised, perhaps indicating astall, or people outside and gathered around the vehicle, includingemergency response vehicles present at the highway grade crossing.

Returning to the example of a person detected in the track 2500 feet inadvance of the train, use cases for the present system are aimed atprotecting human life but none more directly than those involving aperson on the track. Informal polling of experienced locomotiveengineers finds that placing a train in emergency braking application isa rare occurrence with some engineers averaging less than one occurrenceper year. As one would expect, context for these decisions is critical.Like the locomotive engineer, the disclosed system would use itsavailable tools to provide context so that Train Handling and HazardResponse can continually assess the impending severity and applyappropriate train handling. Object Tracking may for instance, determinethe person is crossing the tracks in a perpendicular fashion and willeasily be out of harm's way. The Neural Network may determine the personresponded to the train warning device by waving, looking up or changingcourse. Conversely, the Neural Network may determine that a person islaying prone on the tracks or a small child is on the tracks which wouldtrigger an immediate emergency brake application from Train Handling andHazard Response. This technology allows factors such as these to providecontext allowing the disclosed system to make consistent, best practicesdeterminations from the same visual cues that a locomotive engineer usestoday.

The present system's architecture, which allows its rail specific NeuralNetwork and Object Tracking to focus only on the pixel signature of ahazardous condition and not the entire active view, can provide thecontext required to make these fractional second determinations when ahuman life is at risk. Again, conservation of onboard resources throughthe use of Baseline Navigational Data allows more resources forcontextualizing and updating severity indices for these types ofsituations. Whether a person near the track is waving violently as if tosignal the train to stop or a young child is pulling an imaginary cordto signal the engineer to blow the horn, the present system needscontext and a baseline set of responses from experienced engineers todevelop clearly defined use cases and provide consistent responsesbeyond what is humanly possible in these public safety situations.

The severity index may be mapped to train handling responses. The stateof locomotive and train technology in the freight railroad industryoffers a variety of responses to the unexpected appearance of hazardousconditions along the right of way. Responses may include one or acombination of the following:

-   -   No response to a relatively benign object    -   Ringing the bell    -   Blowing the horn sequence(s)    -   A reduction or elimination of power for propulsion    -   Application of the locomotive brakes (also known as the        independent brake)    -   Application of the dynamic brakes    -   Minimum application of the train brakes (sometimes called the        automatic brake)    -   Mid-range application of the train brakes    -   Penalty application of the train brakes    -   An emergency application of the train brakes

The assignment of risk concept associates a numerical risk with each ofthese or other possible responses. The higher the risk profile, the morerestrictive the action to be prescribed for the train.

FIG. 13 is a flowchart of an exemplary high-level process includingsteps 901-916 by which Train Handling and Hazard Response determinesappropriate actions in response to hazardous conditions in oneembodiment. Those responses include sending commands to the existing EMand PTC systems at steps 913 and 914. EM controls the locomotivethrottle and routine brake applications, while PTC controls Penaltybrake and emergency brake applications which are not recoverable, asonce applied, the train is required to stop prior to a brake systemreset. When a hazardous condition is detected by Active Navigation atstep 901 and is reported to the Train Handling and Response at step 902,Neural Network at step 903 and to Object Tracking at step 904 TrainHandling and Hazard Response assigns a low severity index perhaps onlysounding the horn at steps 907, 908 and the countdown begins with theprospect of assigning the highest severity index. If the Neural Networkfails to classify the hazardous condition within this timeframe which iscomparable to a locomotive engineer's response time. Train Handling andHazard Response will declare the highest severity hazardous conditionand provide PTC with a stop target which includes the distance to thehazardous condition at steps 911 and 913. PTC will then determine if apenalty or emergency brake application is required to satisfy TrainHandling and Hazard Response's stop command. The exception to thisscenario would occur when the braking distance to stop the train is lessthan the distance to the hazardous condition, then Train Handling andHazard Response can allow the Neural Network this additional evaluationtime to classify the hazardous condition before commanding a stop targetto PTC.

Applying FIG. 13 to the exemplary hazardous condition use case of aperson walking in the track detectable at 2500 feet in advance of thetrain begins with Active Navigation declaring an unclassified hazardouscondition which is received by Train Handling and Hazard Response. Ifcurrent braking distance for the train with an emergency brakeapplication is 4000 ft, Train Handling and Hazard Response immediatelysounds the horn at steps 907, 908 and dispatches a drone with thetrapezoidal range and pixel signature of the hazardous condition atsteps 905, 906. The Neural Network then has a prescribed time toclassify the hazardous condition. For this example, the Neural Networksuccessfully classified the hazardous condition as a person walking inthe track with their back to the train. Train Handling and HazardResponse again sounds the horn and Object Tracking confirms the personhas not made a discernable effort to leave the impact zone. In someembodiments, a notification may also be sent to the dispatch center atsteps 909-910 if the severity index is high enough. Train Handling andHazard Response sends a stop command to PTC which places the train in anemergency brake application within the time period established forlocomotive engineer best practices. Train Handling and Hazard Responsecontinues to sound the horn and the dispatched drone advances to thehazardous condition to deliver an audible signal or recorded message inclose proximity to the person as a means of alerting them of theapproaching train.

Restricted Speed operations is another operating scenario addressed byTrain Handling and Hazard Response. PTC limitations can be improved uponwith the present system's capabilities to stop the train within one-halfthe range of vision when a hazardous condition is detected. Whenoperating within a Restricted Speed zone, Train Handling and HazardResponse compares the train's current stopping distance with thedetection range as determined by the trapezoidal range in upcomingBaseline Navigational Data records. Train Handling and Hazard Responsemay then dynamically furnish PTC and/or EM the detection distance and/orsafe operating speed commensurate with the trapezoidal distancesobtained from Baseline Navigational Data, so that the train may beoperated at a speed allowing it to stop short of a hazardous condition,including another train operating on the same track in the oppositedirection at Restricted Speed. The present system can further optimizeRestricted Speed operations by obtaining positive confirmation from adispatch office that no other equipment is in operation on its route.Current industry practice of operating so as to stop in one-half therange of vision could then become obsolete or a rarely used operatingoption. Additionally, drone deployment in a Restricted Speed zoneprojects the present system's pixel calculus screening process andNeural Network beyond the human line of sight, all but rendering thecurrent Restricted Speed practices obsolete.

The present system's Train Handling and Hazard Response may provideinputs to an external communications system onboard the train. Inexemplary embodiments, such an external communication system may becapable of providing prerecorded announcements to persons near thelocomotive and broadcasted on appropriate radio frequencies, such as at160 MHz These messages may include notification of autonomous operations(e.g., informing nearby persons that the train is operatingautonomously), train identification, mile post location, a contact phonenumber of the dispatch center, location of hazmat data for a train, andwarning messages. Warning messages may be notifications relating to theimpending movement of a stopped train, an indication that the train hasapplied its emergency brakes, or that the train needs assistance. Thecommunications system may be able to receive communications from thedispatch center as well to allow for two-way communications with nearbypersons on the track such as railroad personnel or emergency responders.

In some embodiments the present system's Train Handling and HazardResponse may also provision controls and viewing capabilities for remoteoperation from a dispatch center. In the event of an emergencysituation, these controls would allow remote repositioning of the trainat slow speed as requested by emergency responders or railroadpersonnel.

Inventions in accordance with the present disclosure may also addressoperational requirements associated with alarms from wayside equipmentdetectors or incurred by emergency brake applications. In suchscenarios, Train Handling and Hazard Response may deploy anonboard-based drone to perform prudentially or legally mandatedinspections for dragging equipment, hot wheels or bearings and otherdefects providing this information to the present system and to adispatch center with such information as axle counts, bearingtemperatures, pictures, and other pertinent data.

Performance Data and System Analytics

Hazardous condition detection events may be subject to the computerlearning techniques. As seen in FIG. 2 Performance Data and SystemAnalytics collects and flags the data for this analysis. In this way,identification, classification and tracking of hazardous conditionevents are subject to post-tip analysis. After each trip PerformanceData and System Analytics uploads performance data to the presentsystem's lab. Pixel calculus and tolerance values can be tuned toaddress consistent depletion of the tolerance budget or hazardouscondition detection events which fell outside system performanceparameters. This may precipitate adjustments to Baseline NavigationalData such as new pixel calculus values and/or different tolerance valuesfor pixel calculus. A non-limiting example of this would be maintenanceimprovements resulting in changes to the physical plant. Theseimprovements could drive a change in pixels calculus values if shown todecrease the tolerance budget. Similarly, the classification of ahazardous condition which fell outside system performance parameters mayresult in the retraining of Neural Network with image characteristicscollected by Performance Data and System Analytics. Any differencesbetween predicted and actual outcomes as collected by Performance Dataand System Analytics may be modeled and used to further optimize bothsafety and performance of Active Navigation, Neural Network, ObjectTracking as well as the refinement of severity index assignments and theactions of Train Handling and Hazard Response to improve systemperformance standards.

These optimization techniques pair well with the disclosed system's labdevelopment of Baseline Navigational Data. The same suite of tools usedto develop Baseline Navigational Data can be used to optimize itsperformance as well as qualify changes to the present system's computervision software as driven by post-trip analysis of the data collectedand flagged by Performance Data and System Analytics. The presentsystem's lab and data driven approach to computer vision provides theideal environment for regression testing all changes to both the presentsystem's computer vision software and Baseline Navigational Data toensure optimization in one area does not adversely affect another. Asuite of lab regression tests that strategically injects hazardousconditions and interprets system responses with simulated labnavigation, assures the next release of a Baseline Navigational Datafile or the present system's computer vision software contain thedesired improvements without any degradation. The disclosed systemshighly automated approach to the creation and maintenance of BaselineNavigational Data is instrumental in the timely release of both BaselineNavigational Data files and the present system's computer visionsoftware. These releases may be driven by optimization efforts, trackchanges resulting from both planned and emergency work, or even changesto fixed structures in the warning zone.

The present system has a shared consciousness in that all trainsoperating the system benefit from the continuous hazardous conditionrecognition and response evolution. Post trip data is downloaded to acentral back office where automated analysis and testing, as describedabove, drive software enhancements and ultimately new software releases.As different trains traverse the same subdivision over time, datacollected contributes to improvements which include increased objectrecognition data and a more refined system response to objectrecognition data. As a quality control check, improved software is runagainst the data describing actual train runs of the past to verifyappropriate responses are preserved in the proposed software releasebefore being issued. Over time, this methodology of a sharedintelligence ensures a system of increasingly accurate and appropriateresponses to wayside hazards, optimizing the present system's responseto hazardous conditions while improving safety of operations.

Proof of Concepts

An Active Navigation centric Proof of Concepts (POC) was conducted tovalidate the reliability of the present system. The limited scope POCconsisted of drone captured track video with a CMOS camera in which thecritical area (x,y) defined trapezoids were manually established.Subject trapezoids were pixel justified and pixel summation wascalculated for a simple grid superimposed over the impact zone.

FIG. 14 was the baseline image from which Baseline Navigation Data wasestablished.

FIG. 15 represents the image captured by a simulated active locomotiveoperating the disclosed system on a cloud covered day for the samesection of track as in FIG. 14 .

Table 3 contains exemplary data used for the application of ajustification window and the resultant justification value.

TABLE 3 Active Navigation Baseline Navigational Data JustificationWindow 64190 85059 Pixel Sum Justification Window 104.886 138.985 Avg.Pixel Value Active Avg. Justified 138.886 Justification value 34.0997(per-pixel) Active Justified Pixel 84998 Sum

The above justification process successfully adjusted the ActiveNavigation's justification window from a pixel calculus value of 64190to 84998 to closely align with Baseline Navigational Data'sjustification window value of 85059. The per-pixel calculated pixeloffset (the Justification value of 34.0997 in the table above, which isthe difference between the average pixel values for justificationwindows in the Baseline Navigation Data and the Active Navigation data)can then applied to the pixels in the active view before the pixelcalculus values for impact zone and warning zone trapezoids are computedand compared to the pixel calculus values for the correspondingtrapezoids in the Baseline Navigation Data. Applying justification toactive image FIG. 15 provides the resultant image FIG. 16 which is nowjustified and suitable for hazardous condition detection againstBaseline Navigational Data.

FIG. 15 included the introduction of a hazardous condition at 1800 ft.in advance of the locomotive—a white pickup truck. The justified versionof this hazardous condition in FIG. 16 is now compared to BaselineNavigational Data in Table 4 below also included as a reference arevalues for the “no truck present” condition. At 1800′ primarily two gridboxes contain the hazardous condition as represented by the white truck.Functionally, the disclosed system would need to interpret the no truckvalues (4265, 12094) as within tolerance of the Baseline NavigationalData values (5209, 12236). Conversely, the values in Table 4 calculatedby the disclosed system with the truck present (7503, 6647) would needto fall outside tolerance as compared to Baseline Navigational Data(5209, 12236) and be declared a hazardous condition.

TABLE 4 Grid Box 1 Grid Box 2 Pixel Calculus Pixel Calculus BaselineNavigational Data 5209 12236 Justified Active View 4265 12094 No TruckJustified Active View 7503 6647 Truck

While the results of the POC are encouraging, it becomes even moreapparent that current CMOS camera technology is less than optimal forthis application particularly when one considers factors such as fog,snow, or even darkness. While these may be limiting factors for thelocomotive engineer as well, it's obvious more advanced technologies canbring a higher level of safety. Radar, thermal, infrared, LiDAR, or anew advanced technology may provide the present system optimal imaging.Common to any chosen technology however are the benefits of BaselineNavigational Data which provides an unconventional approach to hazardouscondition detection and greatly reduces the burden on a Neural Networkfor object classification which can only prevent a train stoppage withinthe framework of the disclosed system.

Prior Art

Some prior art in the collision detection and avoidance space targetsdriver assist implementations. This prior art is significantly differentfrom the methods and devices described herein because those approachesignore the conservation of onboard computer resources required for fullautonomous operations. None of that prior art demonstrates anyappreciation for the existing PTC, PTL or EM systems in which the RailIndustry is heavily invested. The Rail Industry fully anticipates anyautonomous technology appliance would to be integrated into thesesystems rather than introduce a stand-alone or replacement system.

One prior art patent in the rail vehicle collision detection andavoidance space is U.S. Pat. No. 10,654,499B2, assigned at issuance toRail Vision Ltd. (the “Rail Vision patent”). While the claims in thisRail Vision patent use some similar terminology to describe adriver-alerter system, it cannot reasonably be extended to a platformfor autonomous operations. As such this patent makes no mention of PTC,EM or PTL interfaces which are critical to support autonomousoperations. Moreover, claim 1 of the Rail Vision patent discloses“comparing pre stored images of a section of the rails in front of thetrain with frames obtained during the travel of the train in order toverify changes in the rail and in the rail's close vicinity; anddetecting obstacles based on this comparison.” These image subtractionstrategies are in sharp comparison to the current disclosure, which doesnot directly compare current images and historical images to each otherbut rather calculates current image pixel calculus values and comparesthem to corresponding pixel calculus values in the Baseline NavigationalData.

Moreover, the Rail Vision patent's zone of interest (16.5) differsmarkedly from the disclosed systems impact zone and warning zone in bothits acquisition and utility. The zone of interest is discovered in eachimage using IR technology and comparing temperature differences betweenthe rail and its surrounds. Conversely the disclosed system uses dataembedded in the Baseline Navigation Data file to provide (x,y)coordinates to Active Navigation to define both the impact zone and thewarning zone. These off-line derived trapezoidal references conserveonboard resources and provide the utility of trapezoidal ranging,comparatively the prior art employees triangulation methods, Section17.5, to establish distance to an “object”.

CONCLUSION

The infrastructure associated with various rail routes, rail crossings,connecting tracks, draw bridges and so forth with multiple routes can becomplex. There is the added complexity when new rail is being installedbecause at times replacement strands of steel rail are positioned forinstallation within, or just outside the two installed rails or “gaugeof the track”. Whether locomotive or drone image collection devices arein use, focusing system resources only on areas of interest through datadevelopment differentiates the present systems effectiveness as opposedto conventional approaches. As one non-limiting example, it should beunderstood the concept of focusing system resources only on areas ofinterest may be used in alternative embodiments that may not use all ofthe concepts and techniques disclosed herein. For example, the techniqueof inputting only a pixel signatures (rather than an entire image froman image collection device) to a classification scheme may be usedregardless of whether the technique for identifying the pixel signatureis performed using one of the techniques disclosed herein or analternate technique.

Several embodiments of the present disclosure have been described. Whilethis specification contains many specific implementation details, theseshould not be construed as limitations on the scope of any disclosuresor of what may be claimed, but rather as descriptions of featuresspecific to embodiments of the present disclosure.

Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination or in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented incombination in multiple embodiments separately or in any suitablesub-combination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults.

Moreover, the separation of various system components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order show, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous. Nevertheless, it will be understood thatvarious modifications may be made without departing from the spirit andscope of the claimed disclosure.

What is claimed is:
 1. A method for autonomously detecting onboard atrain a potential hazard condition on a railroad track, the methodcomprising: obtaining a position of the train on the track; accessingbaseline navigational data for each of a plurality of regions of trackin advance of the position of the train, the baseline navigational datafor each of the plurality of regions including coordinates defining theregion and at least one historical pixel calculus value calculated basedon pixel values of historical image data corresponding to the region,the historical pixel calculus values in the baseline navigational datahaving been calculated prior to a current trip of the train; accessing acurrent image of the plurality of regions of track in advance of theposition of the train; justifying the current image; calculating atleast one current pixel calculus value for each of the plurality ofregions based on pixel values of the justified current imagecorresponding to the region; performing, for each of the plurality ofregions, a comparison of a tolerance value to a difference between theat least one current pixel calculus value and the at least onehistorical pixel calculus value for the region. determining a potentialhazard condition based on at least one of the comparisons.
 2. The methodof claim 1, wherein the historical pixel calculus values are calculatedby performing a weighted summation of the pixel values of the historicalimage data corresponding to the region, and wherein the current pixelcalculus values are calculated by performing a weighted summation ofpixel values of the justified current image corresponding to the region.3. The method of claim 2, wherein the baseline navigational data for atleast one of the regions further includes definitions for a plurality ofgrids that divide the region and historical pixel calculus values foreach of the plurality of grids based on pixel values of a portion ofhistorical image data corresponding to a respective grid; and whereincalculating the at least one current pixel calculus value for the atleast one of the regions comprises calculating current pixel calculusvalues for each of the plurality of grids based on pixel values of aportion of the justified current image data corresponding to arespective grid.
 4. The method of claim 2, wherein justifying thecurrent image comprises: accessing, for at least one justificationwindow, a baseline justification value based on a summation of the pixelvalues of historical image data corresponding to the at least onejustification window; calculating, for the at least one justificationwindow, an active view justification value based on a summation of thepixel values of the current image data corresponding to the at least onejustification window; applying a justification factor based on adifference between the baseline view justification value and the activeview justification value to pixel values in at least portions of thecurrent image corresponding to the plurality of regions.
 5. The methodof claim 4, wherein the at least one justification window comprises aplurality of justification windows.
 6. The method of claim 4, whereinthe justification value is a per-pixel justification value that isfurther based on the number of pixels in the at least one justificationwindow.
 7. The method of claim 2, further comprising the step of:transferring the justified image data for any region corresponding to adetected hazard to a classification process for classifying an object inthe region corresponding to the detected hazard, the classificationprocess being configured to process only regions of the justifiedcurrent image corresponding to a potential hazard condition.
 8. Themethod of claim 7, wherein the classification process is optimized toclassify human beings, highway vehicles, animals, railcars, locomotives,trees, poles, rock slides, washouts, and misaligned track.
 9. The methodof claim 7, further comprising: in response to the classificationprocess being unable to classify the potential hazard condition within apredetermined time period or with a predetermined certainty, assigning ahighest severity index to the hazardous condition.
 10. The method ofclaim 2, further comprising the step of tracking movement of an objectcorresponding to a potential hazard condition.
 11. The method of claim2, further comprising: accessing, in the baseline navigation data, abaseline unobstructed distance corresponding to the position of thetrain, the baseline unobstructed distance representing a distance to apotential obstruction as measured by reflective technology prior to acurrent trip of the train; and accessing, from reflective technologymounted on the train, a current unobstructed distance representing adistance to a potential obstruction as measured by reflectivetechnology; wherein the step of determining a potential hazard isfurther based on a comparison of the baseline unobstructed distance andthe current unobstructed difference.
 12. The method of claim 2, whereinthe further comprising the steps of: determining that the train is in amulti-track location; receiving an indication that an other rail vehicleis present on a nearby track; and reducing the number of the pluralityof regions of track in advance of the position of the train for whichbaseline navigation data is accessed to compensate for a reduction invisibility resulting from the presence of the other rail vehicle on thenearby track.
 13. The method of claim 2, wherein the track in advance ofthe position of the train is divided into a plurality of sections, andplurality of regions includes an impact zone and warning zones on eachside of the impact zone for each section of track.
 14. The method ofclaim 13, further comprising the step of issuing a warning when ahazardous condition is detected in a warning zone.
 15. The method ofclaim 13, further comprising the step of issuing a warning when ahazardous condition is detected in an impact zone.
 16. The method ofclaim 13, further comprising the step of issuing an indication that thetrain's brakes should be activated in response to a detection of ahazardous condition that corresponds to a condition requiring brakeactivation in an impact zone at a distance which is substantially equalto or less than a distance in which it is possible to stop the train.17. The method of claim 2, wherein the current image is obtained from asensor mounted on the train.
 18. The method of claim 2, wherein thecurrent image is obtained from a sensor that is not mounted on thetrain, the sensor being selected from the group consisting of a sensormounted on a drone and a sensor mounted in a fixed location on a trackwayside.
 19. The method of claim 2, wherein the sensor is an imagecollection device configured to image light in the visible spectrum. 20.A method for developing baseline navigation data comprising: determiningthat a rail vehicle has traveled a fixed distance along a length oftrack; in response to the determination, capturing an image from acamera mounted to a rail vehicle and determining an unobstructeddistance ahead of the rail vehicle using a reflective technology sensor;detecting an unobstructed distance in the absence of any hazardouscondition using reflective technology at each location corresponding toan image; defining, for each image, a coordinates for a plurality ofregions of track in advance of the position of the rail vehicle;calculating, for each region in the image, a pixel calculus value;calculating, for at least one justification window in the image, a pixelcalculus value; identifying at least one pixel for at least one rail inthe image; storing, for each image, a record comprising the pixelcalculus value for each of the plurality of regions in the image, thecoordinates of each of the regions in the image, the pixel calculusvalue for the at least one justification window in the image, alignmentdata comprising coordinates for the at least one pixel of the at leastone rail, and the unobstructed distance corresponding to the location ofthe image.